Repository navigation
FEAT: Save converter copies (part 18) - #4069
varunj-msft wants to merge 547 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Synchronous discovery and component construction in restore() can stall health and configuration requests during live apply and should be offloaded while preserving initialization that requires the runtime loop.
1 open issue
What changed in this PR
Adds persistent converter copies to PyRIT’s registry, building on the stacked persistence and connection-setting safeguards. Saved copies retain only their overrides and are rebuilt from their source after restart.
Changes:
- Adds saved-entry recipes, restore handling, origin metadata, and safeguards against replacing saved entries.
- Makes converter-copy saving the default in the dialog, with a Temporary fallback and name validation.
- Extends registry displays, documentation, and regression coverage, while isolating test recipe storage.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging safeguards. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement refusal. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe storage configuration. |
| tests/unit/registry/test_resolution.py | Tests copy-locked parameter metadata. |
| tests/unit/registry/test_instance_registry.py | Tests saved-entry protection and origins. |
| tests/unit/registry/test_converter_registry.py | Tests collisions and locked copy settings. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/models/test_parameter.py | Updates parameter serialization expectations. |
| tests/unit/backend/test_temporary_components.py | Tests target-copy connection safeguards. |
| tests/unit/backend/test_technique_edit_service.py | Tests locked converter settings in edits. |
| tests/unit/backend/test_scorer_routes.py | Tests scorer persistence and API errors. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch conflict messages. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage responses and logs. |
| tests/unit/backend/test_saved_entries.py | Tests temporary-reference and credential refusals. |
| tests/unit/backend/test_mappers.py | Tests target origin and saver metadata. |
| tests/unit/backend/test_initializer_service.py | Tests local storage error sanitization. |
| tests/unit/backend/test_auth_middleware.py | Tests current-user ID extraction. |
| tests/unit/backend/test_analytics_benchmark.py | Tests isolated benchmark recipe storage. |
| tests/unit/backend/conftest.py | Isolates saved entries in backend tests. |
| tests/end_to_end/conftest.py | Uses temporary backend recipe storage. |
| test_support/analytics_benchmark.py | Configures benchmark recipe isolation. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry substitutions. |
| pyrit/setup/initializers/targets.py | Protects saved names during target grouping. |
| pyrit/setup/initializers/scorers.py | Prevents tagging saved substitutes. |
| pyrit/setup/configuration_loader.py | Adds recipe storage configuration. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares transcription credential restrictions. |
| pyrit/score/scorer.py | Adds scorer persistence policy metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares transcription persistence restrictions. |
| pyrit/registry/resolution.py | Derives copy-locked parameter metadata. |
| pyrit/registry/registry.py | Rejects locked external copy overrides. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Preserves metadata and detects saved collisions. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares target persistence restrictions. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/openai/openai_target.py | Declares persistence policies and locked settings. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Prevents saving credential-bearing HTTP requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Adds shared persistence and copy policies. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and destination policies. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares SAS and destination policies. |
| pyrit/prompt_target/a2a_target.py | Declares destination validation metadata. |
| pyrit/models/parameter.py | Adds the copy-locked parameter flag. |
| pyrit/models/catalog/target.py | Adds target origin and saver fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin metadata. |
| pyrit/models/catalog/registry_recipe.py | Defines saved recipes and restore reports. |
| pyrit/models/catalog/__init__.py | Exports recipe and restore models. |
| pyrit/converter/converter.py | Adds shared copy and credential policies. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Locks speech connection settings. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Locks speech connection settings. |
| pyrit/backend/services/scorer_service.py | Saves scorer recipes and reports origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes safe saved-entry conflict messages. |
| pyrit/backend/services/scenario_run_service.py | Detects launch-script saved-entry conflicts. |
| pyrit/backend/services/runtime_lifecycle.py | Restores recipes and reports readiness failures. |
| pyrit/backend/routes/targets.py | Passes saver context and maps persistence errors. |
| pyrit/backend/routes/scorers.py | Supports scorer persistence responses. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage failure logging. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage failure logging. |
| pyrit/backend/routes/initializers.py | Handles and sanitizes storage failures. |
| pyrit/backend/routes/converters.py | Maps save errors and blocks saved removal. |
| pyrit/backend/routes/common.py | Adds saver and save-refusal helpers. |
| pyrit/backend/README.md | Documents persistence APIs and converter copies. |
| pyrit/backend/models/targets.py | Adds target persistence request validation. |
| pyrit/backend/models/scorers.py | Adds scorer persistence request validation. |
| pyrit/backend/models/converters.py | Adds converter persistence and origin fields. |
| pyrit/backend/middleware/auth.py | Exposes the current user’s object ID. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver metadata. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin metadata. |
| frontend/src/utils/registryOrigin.ts | Defines shared origin labels. |
| frontend/src/utils/registryName.ts | Generates valid, available registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence and origin API types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and saver test data. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal detection. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save expectations. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Displays scorer origin badges. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests scorer origin labels. |
| frontend/src/components/Registry/OriginBadge.tsx | Adds a shared registry origin badge. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Shows origins and disables saved removal. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origin labels and removal restrictions. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Saves copies with Temporary fallback. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests copy saving, refusals, and names. |
| frontend/src/components/Config/TargetTable.tsx | Shows origins, savers, and ownership filtering. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests origin and saver displays. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles saver labels and filter controls. |
| frontend/src/components/Chat/ConverterPanel.save.test.tsx | Expects persistent converter-copy requests. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline fixture recipe storage. |
| frontend/e2e/config.spec.ts | Tests target saving and saver filtering. |
| frontend/e2e/_targets.ts | Adds origin and saver mock fields. |
| doc/gui/0_gui.md | Explains saved entries and converter copies. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restore failures. |
| doc/code/registry/0_registry.md | Documents locked copy connection settings. |
| .pyrit_conf_example | Adds commented recipe storage configuration. |
Review effort: Balanced
| # the target API builds them, because a constructor such as HuggingFaceChatTarget's starts a task | ||
| # on it. A slow constructor holds the loop while it runs. | ||
| saved = await asyncio.to_thread(restorer.read) | ||
| report = restorer.restore(saved=saved) |
There was a problem hiding this comment.
Leaving this as it is. The restore code comes from #3909 and builds entries on the loop on purpose: a constructor such as HuggingFaceChatTarget's starts a task on the running loop, and the target API builds them the same way. Reading the files already runs in a worker thread. A saved copy doesn't change that, since the converter it copies was built the same way at startup.
# Conflicts: # pyrit/backend/middleware/error_handlers.py # pyrit/backend/routes/scenario_presets.py # pyrit/backend/routes/scenario_suites.py # pyrit/registry/file_document_storage.py
…y the other storage tests do
# Conflicts: # pyrit/backend/README.md
# Conflicts: # pyrit/backend/README.md
… its Entra ID default
… a saveability answer once its request goes away The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
…s, and auth mode only
Signed-in users may create shared converters and scorers that connect with this server's credentials, so a saved one isn't refused for that. This takes out the refusals of the Azure Speech converters and the Content Safety, audio and local refusal scorers. A saved entry still can't hold a typed key.
# Conflicts: # pyrit/backend/models/converters.py # tests/unit/registry/test_registry_recipe_rules.py

Description
This is one of the changes for saving Registry entries across restarts. It builds on #3984 (merged), which keeps a copy's endpoint and credentials the same as its source's. Until the earlier PRs merge, the diff here also shows their changes.
Save to registry on a converter's changed settings, in chat or in a technique edit, made a Temporary copy that was gone after a restart. Copies can now be saved, and a saved copy is rebuilt from the converter it copies at every start.
sourceand"persist": trueis saved as a recipe that names the converter it copies and holds only the changed settings. At every start it's rebuilt from that converter as it is then, after it, so it keeps where that converter sends requests and how it signs in.What stays the same: a copy made with Create as Temporary is Temporary, as before, and a copy still can't change its source's connection settings (#3984). Target copies are unchanged.
Screenshots: Save to registry for a copy of a saved converter, the saved copy in the Converters table, and a copy of a Temporary converter offered as Temporary with the reason.



Not covered:
sourceon targets) and gives copies per-caller credentials.